Unlike other engineering disciplines, software engineers have little in the way of quantitative information that they can consult when making design decisions. There are no omnibus catalogs of Application Programming Interface (API) performance costs. In this instance, the term “API” refers to a single function as in “The InvalidateRect API” rather than a family of functions as in “The Windows® Operating System API” available from Microsoft® Corporation in Redmond, Wash.
Additionally the situation is complicated by the fact that most APIs cannot have their cost characterized by a single number. Cost can be measured in many different ways—such as memory usage, CPU usage, I/O, costs and so forth and it can be dependent on input parameters as well as context and of course available hardware. As a result there is generally no attempt made to characterize the performance of APIs at all. This leaves engineers in the unfortunate position of having no a priori guidance available when making design decisions. They must resort to prototyping or worse yet, simply guessing, ignoring cost considerations entirely, or just hoping for the best. These latter options are all too common. Thus, the collection of API data is very desirable to provide useful knowledge of API performance or resources costs.
It is therefore desirable to gather highly accurate memory allocation and execution time data for API operation. One objective may be to provide the data to API consumers such that they can make informed decisions about whether the particular API components have memory and timing characteristics that are reasonable for their intended use. Since both allocation and timing will vary depending upon how the functions are used, the data preferably may reflect the statistical distribution of allocation and timing across a broad range of real-world scenarios. This distribution can constitute a ‘performance profile’ which may be used both to troubleshoot an API's behavior as well as to document these characteristics to consumers.
Although memory allocation may be measured with nearly 100% accuracy, measuring execution time is fraught with uncertainty. The measurement itself takes up a portion of the overall execution time which then distorts the data. This is particularly true when gathering the timing characteristics of a set of interdependent functions in a single profiling run. The measurement overhead for function calls deeper in the call tree will accumulate and alter the results for functions higher in the call tree. For example, if function A calls function B thousands of times in a tight loop and we are measuring the timing of A and B in the same run, then the cumulative measurement cost for B could dramatically alter the results for A. Existing tools attempt to address the issue by subtracting away the cumulative measurement error, but this approach fails to yield statistically significant results.
Generally, existing API measurement methods are used in the context of profiling tools which can record detailed cost information about a particular execution. These systems however are not suitable for omnibus data gathering because the act of measuring itself perturbs the cost. They are designed to harvest as much information about the whole program as possible in a single run. It is desirable to gather information about particular APIs over a variety of executions. The present invention addresses these and other concerns.